lh fix individual read deduplication and limiting #195
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Owner
lhotari
commented
Nov 19, 2024
lhotari
commented
- Add solution to PulsarMockBookKeeper for intercepting reads
- Add validation to getManagedLedgerMaxReadsInFlightSizeInMB
- Don't divide by 0 if it can be avoided
- Estimate entry size for InflightReadsLimiter by keeping stats
- Limit replay messages by bytes size
- Test limiting by bytes size
- Test
- Avoid exception which commonly happens
- Add debug logging to InflightReadsLimiter
- Clear the cache in the test
- Add logging
- Fix read limits for individual reads
- Revert "Test"
- Reduce timeout
- Apply cap for delayed messages too
- Improve asyncReplayEntries
- Refactor to have a single replay method
- Optimize filtering deleted messages
- deprecate synchronous replayEntries method
- move asyncReplayEntries without sortEntries as a default method
- Perform replay reads in ranges
- Make test to fail
- Fix disabled timeout in InflightReadsLimiter
- Add queuing to InflightReadsLimiter when the limit is reached
- revisit
- Ensure that cached entry readerIndex is not tied to the original one
- Adjust InflightReadsLimiterTest
- Add timeout executor to InflightReadsLimiter
- Polish
- Fix test
- Instantiate ArrayList directly
- Support caching replayed entries
- Start addressing the removal issue with a removal queue
- Move EntryWrapper to upper level
- Move RangeCache to cache package
- Start adding removal queue
- Move towards adding removal queue changes
- More moves towards removal queue solution
- Handle removing by size in scheduled task so that blocking would be rare
- Remove unused code
- Update RangeCacheTest with removal queue
- Fix imports
- Add license headers
- Handle all cache evictions by the same thread
- Move RangeCacheRemovalCounters to top level
- Use removal queue in RangeCacheTest
- Handle eviction
- Handle adding atomically to removal queue and cache
- Improve javadoc
- Use unbounded queue for removal queue since cache size is bounded by size
- Refactor eviction when cache size exceeds evictionTriggerThreshold
- Start adding cacheEvictionByExpectedReadCount
- Add CachedEntry
- Activate cursor when consumers disconnect and connect
- disable checkCursorsToCacheEntries when eviction by read count is enabled
- Always keep cursors with connected consumers in "active" state when eviction is based on expected read count
- Add a way to find cursors before current cursor
- Pass predicate for deciding whether to cache or not
- Reduce coupling to EntryImpl
- Extract abstract base class to be used for Entry implementations
- Introduce CachedEntry for eviction by read count implementation
- Remove generics type parameters from RangeCache since there's no need to reuse the class with different types
- Add TODO about the solution for handling skipping of the entries
- Add note of replacing the use of peek
- update design
- Evaluate entry size when it gets added
- Fix issue in getting the entry
- Fix RangeCacheTest
- Remove generics
- Add simple stash
- Add note
- Fix test that expects old defaults
- Adapt test to new behavior
- Improve removal queue
- Fix EntryCacheManagerTest
- Support configuring ManagedLedgerConfig defaults for tests
- Fix ManagedLedgerTests
- Use waitForPendingCacheEvictions in EntryCacheManagerTest
- Use Double.NaN instead of Double.POSITIVE_INFINITY
- Start adding expected read count solution
- Improve ManagedCursorContainerTest
- Implement getNumberOfCursorsAtSamePositionOrBefore
- Implement EntryReadCountHandler
…ngUtils#randomAlphanumeric (apache#23645)
…ther users have published TXN messages in before. (apache#23210) [improve] [pip] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before. (apache#23210)
…e to the container (apache#23657)
…n shaded Java client modules (apache#23647)
…n creating retryLetterProducer has failed (apache#23560)
…n consumer creation (apache#23254)
…tionProvider to avoid exception thrown (apache#23663)
This reverts commit 3ae9991.
… to reuse the class with different types
lhotari
force-pushed
the
lh-fix-individual-read-deduplication-and-limiting
branch
from
46dbb23
to
f58ad8c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.